Record: 5-expert Hedge Mixer + CROWN-Q + stride=64 (val_bpb=1.0541)

val_bpb: 1.0541 (3-seed mean) | ~15.7 MB | 8xH100 SXM

Results (8xH100 80GB SXM)

Contributions

1. CROWN-Q Training Penalty (training-time)

Added a quantization-aware penalty during warmdown that penalizes weights sensitive to quantization error:

crown_q_loss = lambda * mean(w^2 * delta^2 / 12)

where delta = row_max / clip_range is the per-row quantization step size. This encourages weights to be quantization-friendly, reducing post-quantization degradation. CROWN_Q_LAMBDA=0.01.

Effect: Slightly better compression (artifact ~200KB smaller) and more robust quantization.

2. Eval stride 32 -> 64 (eval-time)

Changed sliding window stride from 32 to 64 during evaluation. Experiment showed identical BPB quality but 2x faster scoring. Frees ~100s of eval budget for more TTT epochs.

3. TTT Epochs 3 -> 4 (eval-time)

Increased test-time training from 3 to 4 epochs per chunk, using the time freed by stride=64. Each additional epoch adapts the model more to scored data. Tested 8 epochs but that overfits (1.0735 vs 1.0473 for 4 epochs).

Combined Effect

• stride=64 saves ~100s of eval time
• 4th TTT epoch uses ~85s of the saved time
• Net eval time: ~336s (down from ~562s), well within 600s budget
• BPB improvement: 1.0745 -> 1.0541 (-0.0204)

Architecture

Reproduction

DATA_PATH=../data/datasets/fineweb10B_sp1024 \
TOKENIZER_PATH=../data/tokenizers/fineweb_1024_bpe.model \
SEED=1337 MAX_WALLCLOCK_SECONDS=600 \
USE_MIXER=1 MIXER_ETA=0.1 \
TTT_EPOCHS=4 TTT_FREEZE_BLOCKS=2 \
TTT_LR=0.0001 TTT_CHUNK_TOKENS=131072 \
ADAPTIVE_LR=1 ADAPTIVE_LR_MAX=3.0 \
EVAL_STRIDE=64 \
CROWN_Q_LAMBDA=0.01 \
torchrun --standalone --nproc_per_node=8 train_gpt.py

Compliance

Credits

• Base model: PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 by @signalrush
• TTT recipe: PR Non-record: 11L Depth Recurrence + High-Yield Legal TTT (1.14458 BPB) #461 by @Christopher-Lee-McClendon
• CROWN-Q concept: PR Record: CROWN-Q + Full GPTQ + SWA/EMA Blend — val_bpb 1.1186 (3-seed mean) #693 by @EthanYangTW
• 5-expert Hedge mixer: PR Record: 5-expert Hedge Mixer + TTT (3-seed mean val_bpb=1.0745) #688